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ABSTRACT 

Five causal models relating several aspects of 
end-of-terai student evaluation of a graduate course in nursing 
research methods were proposed and tested empirically. The £o a ^2 a 
evaluation form consisted of four Likert-type subscales, on vtH? h 
students rated the following aspects of the course: (1) the e*\e a t tD 
which the course met its objectives; (2) the utility of vai:io a % 
instructional resources and activities in meeting course ot>3e£U^5 
(e.g., reading materials, quizzes, examinations); (3) the 
effectiveness of a number of specific course policies (e.g, • Pfce- 1 
book examinations and quizzes) in promoting learning: and (4) 
instructor effectiveness. For each model, a prediction was ^a^e iti 
terms of vanishing partial or zero-order correlations. The %o&\ 
plausible models suggested that ratings of the extent to wftict* course 
objectives were met, and of instructor effectiveness, were 
linked with ratings of course policy effectiveness via ratt^g^ fc** e 
utility of instructional resources in meeting course objective, 
Course policies may best be implemented by means of appropriate 
instructional resources and techniques: students will then ^a fc e t he 
instructor as effective and regard course objectives as ha<An£ p a sn 
met. (Author/RL) 
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A..:3trac7, 

• ■■ — k 

Five causal models relating several aspects of stuu^v - 
evaluation of a graduate course in cursing research methods vere 
proposed and tested empirically * Thj most plausible models suggested 
-hat ratings of the extent to which course objectives were met, and 
of instructor effectiveness, were both linked with ratings of course 
policy effectiveness via ratings of the utility of instructional 
resources in meeting course objectives. Course policies may best 
be implemented by means of appropriate instructional resources and 
techniques? students will then rate the instructor as effective and 
regard course objectives as having been met. 
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a) effectives 

The purp of the preae" study wax to test alternative 
causal models rearing se^er^l =sreets of nosing students 1 eval- 
uations of a graduate course in :rars~ng research meuaods. 

b) Per3pectives,r " arjt ical graiBewor^ 

Although cause-li^y can ne^er be iir.sstly inferred from 
correlational data, causal inferences maj- be made concerning the 
adequacy of specific causal ^odel. (Simon, --57; Blalcck, 1964; 
Duncan, 1975) • The methcd for mat, such inferences involves 
the explicit definition of a iir^ :e set of variables, assumptions 
about the causal connections amozt -those variables, and assumptions 
about the effect of outside variables upon the variables in the model. 
Models are then eliminated which make predictions which are incon- 
sistent with the data. The conditions imposed on each model are 
the predictions expressed in terms of the correlation coefficients 
(either zero-order or partial) that should be obtained if the 
model is correct . These predictions are always in terms of vanish- 
ing partial or zero-order correlation coefficients (Nachmias and 
Nachmias, - 1976) • 

In addition to the agreement between predicted and obtained 
correlation coefficients, the temporal order among the variables 
must be taken into account in establishing causation. In responding 
to an evaluation questionnaire at the end of a course, students 
are typically asked to give a summary rating of a variety of 
different aspects of the course — objectives, materials, lectures, 
instructor effectiveness and so on. Although it may be difficult 
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to establish a temporal order among these several aspects of the 
course at the time of evaluation, it is quite reasonable to 
assume that some causal relationship exists between them, For 
instance, instructor competence, or lack of it, may influence 
the framing of course objectives. On the other hand, poor objectives, 
or inadequate instructional materials, may render an otherwise 
capable instructor ineffective in the eyes of his or her students, 
These causal connections may, in turn, be reflected in student 
ratings, 
c) Methods , 

Data from student evaluations of a masters 1 level course 
in nursing research methods were subjected to correlational anal- 
ysis as follows. (See Appendix for a copy of the evaluation form, 
along with means and standard deviations of student ratings of each 
item), The course evaluation form consisted of four Likert-type 
subscales, on v/hich students rated these aspects of the course; 
the extent to which the course met its objectives (variable 
the utility of various instructional resources and activities 
in meeting course objectives (e.g., reading materials, quizzee, 
examinations) (variable ty\ C\t effectiveness of a number of 
specific course policies (e.g., opsn book examinations and quizzes, 
team research projects, critiquing published research in class) in 
promoting learning (variable and instructor effectiveness 
(variable X^), Each subscale consisted of between eight and eleven 
items. Subscale reliabilities (internal consistency as indexed by 
Cronbach'B alpha coefficient) are shown in Table 1. 



Table 1. Subscale Reliabilities (Cronbach's Alpha Coefficient! 
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Subscale 

1. Extent to which course met 
objectives 

2. Utility of instructional re- 
sources and activities 

3. Effectiveness of specific course 
policies 

4. Instructor effectiveness 



Reliability Coefficient 
0,79 

0,83 

0.14 

0,83 



The matrix of intercorrelations among the four subscales is shown 
in Table 2 below. 

Table 2. Intercorrelations Among Evaluation Form Subscales 



* 2 



.52 .35 ,51 
- .53 ,51 
- .47 
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Jive causal models suggested by the work of Goldberg (1966) and 
Rehberg, Schafer and Sinclair (1970) were hypothesized and tested. 
The first three were three-variable models involving aspects of 
the course apart from instructor effectiveness. 



Model I 



Model II 



Model III 



i 
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Two mor -: -nodels invol"±ag all four variables ware te~s-.: next. 
;odel IV Model V 



\ 

^3 — X 2> 





d) Data Source 

End-of- ~erji stiadent evaluations of a masters' L™el course 
in nursing reieHrch methods at a large eastern university were the 
data source for ~h_is study. The sample consisted of 10- students 
in three section, all taught by the author, Confidentiality of 
responses was iz^ured by the fact that students wtre asked not to 
identify themselves in any way. 

e) Results 

For each model, a prediction v/as made in terms of vanish- 
ing partial or zero-order correlations. Predictions and empirical 
results for the five models are shown in Table 3 °n page 5. 
Of the three- variable models, Models I and III can be eliminated 
as implausible. Model II fits the empirical results fairly well, 
and hence, may be supported. Although neither Model IV nor Model V 
provides a close fit to the data, Model V appears to be the more 
plausible of the four-variable models. In summary, Model II 
suggests that students 1 perception of the utility of instructional 
resources in meeting course objectives intervenes between ratings 
of course policy effectiveness and of the extent to which course 



Tat'le 3. Predictions and Em- irical Results for Models T - V 

Preelections Em pirical Results 

Model I r 1£ . 0 r^^ = , 42 2 

Model II r r , , . 0 r i". 2 88 « 096 

Model III r ; - =0 r 23 = .531 

Model IV . e3 = 0 r 24>3 = .439 

~ -24 = 0 r 13.24 = .027 

Model V r>, 2 =0 r 34.2 = * 238 

r ,24 = 0 r i 3 ! 2 4 = '° 27 



objective are net. When a fourth variable (instructor effectiveness) 
is added - : the model, as in Model V, the basic structure of 
Model II ;L3 still discernible — but now, ratings of both instructor 
effectiveness and of the extent to which objectives are met are 
linked with ratings of course polic y via the intervening variable, 
perception of the utility of instructional resources. It may be 
that course policy is best implemented by means of appropriately 
chosen instructional resources and techniques: in which case students 
will rate the instructor as effective and regard the course objectives 
as having teen met. 

f). Educational/Scientific Importance 

A number of recent studies (e.g., Tetenbaum, 1975; Critten- 
den and Norr, 1975; Prey, Leonard and Beatty, 1975; Scott, 1975; 
Whitely and Doyle, 1976; and Marsh, 1977, 1980) have addressed the 
problem of the reliability and validity of student ratings of in- 
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struction. However, once these have been established, the question 
of practical use in the improvement of instruction remains. What 
are the relationships among the various aspects of rtudent evaluation 
of instruction? Wha~ variables can or should be man±pt lated in order 
to improve the quality of instruction? In view of tr.e difficulty 
of conducting true experiments in order to answer zi--se questions, 
other, less direct approaches to exploring causal relationships 
must be considered. Although the present study involved ^ fairly 
small sample of students and relatively simple models, results 
suggest that the causal modeling approach may be c: considerable ralue. 
A thoughtfully constructed causal model, empirically validated, can 
identify those variables in a given causal sequence whose manipulation 
is most likeiy to yield change in a desired direction. The almost 
universal practice of having students evaluate each course at its 
conclusion should provide a large and rich source of data for 
further exploration of the potential of this technique for assessing 
and improving instruction. 

However, causal modeling should not be embraced with un- 
critical enthusiasm. Magoon (1976) has pointed out that enduring 
problems plague the application of causal modeling in a number of 
research settings. Not the least of these problems is the fact that 
researchers must always work with measuring tools which are less than 
perfectly reliable. The author of the present paper felt that it 
might be worthwhile to investigate, in a purely speculative spirit, 
whether significant changes would result in the interpretation of the 
results if all the correlations between the subscales of the evaluatior 
tool were corrected for attenuation (Nunnally, 1978). Measurement 
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error — unreliablLi" — makes correlations between measures less 

than they would be i± measurement error were not present. Fro:- 

the theory of measurement error one may estimate how much correlations 

between true scores vculd be higher than those between fallible 

scores. If there were no error present, the correlation r 1 ^ 

between two measures v/ould be given by 

r . 
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where r^ 2 is the correlation between the fallible measures X 1 
and Xg, and r^ and ate the reliabilities of X 1 and X 2 > respec- 
tively. The equation above should really be regarded as an estimate 
rather than a correction — it is an estimate of what the correlation 
between two measures would be if these measures were perfectly 
reliable. The results of applying the correction for attenuation 
to subscale intercorrelations are shown in Table 4 below. 



Tabl e 4, » Intercorre l ations Among Evaluation Form Subscales Corrected 
for Attenuation 

^2 ^3 ^4 



X 1 - .64 .59 .63 

X 2 - « .88 .70 

X 3 - . .78 

X - - - - 



The zero-order and partial correlation coefficients in Table 

3 were recalculated, using the values in Table 4. The results are 
shown in Table 5. 
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Table 5. Predictions and Empirical Results for Models I - V Based 
on Subscale Intercorrelations Corre cte d for Attenuation? 

Predictions Empirical Results 



Model 

l 1 --s \JL Ks -i- 


I 


r 12.3 


0 


T* — 

12.3 


.315 


Model 


II 


r 1 3 . 2 = 


0 


r 13.2 


.083 


Model 


III 


23 


o 


23 


.880 


Model 


IV 


r 24.3 = 


0 


r 24.3 = 


.C60 






r 13.24 " 


0 


r 13.24 = 


-.091 


Model 


V 


r 34.2 = 


0 


r 34.2 


.480 






r 13.24 = 


0 


r 13.24 


-.091 



Of the three-variable models, Model II still appears to be the 
best supported by the data as before. However, of the four- variable 
models, Model IV clearly appears to be the more plausible, rather 
than Model V. Model IV suggests that student perceptions of the 
extent to which course objectives were met is linked with the 
effectiveness of specific course policies via ratings of instructor 
effectiveness and the utility of instructional resources and ac- 
tivities. The interpretation ch;,oiges somewhat: course policy may 
best be implemented by means of appropriately chosen instructional 
resources applied by an effecti v e instructor, in which case students 
.will judge that the course objectives have been met. 

As Nunnally points out (1978) there is some controversy 
regarding the correction for attenuation: one may deceive one's 
self into believing that a better correlation has been found than 
it? actually warranted by the data. In addition, it is often 
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argued that the correction for attenuation provides a poor estimate 
of the correlation actually obtained between variables when they 
are made highly reliable. 

If the preceding discussion does nothing else, it should 
suggest that if the causal modeling approach to analysis of course 
evaluation data ib to be taken seriously as a guide to the improve- 
ment of instruction, a critical ingredient in the process is a 
course evaluation tool with highly reliable subscales .Efforts must 
be directed toward the development of such tools, since student 
course evaluations are becoming an ever more important consideration 
not only in the improvement of instruction, but in the making of 
administrative decisions regarding promotion and tenure of faculty. 
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UNIVERSITY OP MARYLAND 
SCHOOL OP NURSING 

Nursing 701 - Course Evaluation 

Section 1 « On a scale (Likert) of 1 to 5, please assess ho v v/^]^ 
the course met its objectives* 1 = not at all; 5 = compleTfT^^^ 
met its objectives. 

X S.D. 



4.39 


0.41 


1 . 


Applies the components of the research j>f°G&^$ 
in nursing. 


4.46 


0.65 


2. 


Explains appropriate statistical procedures 
estimating the reliability and validity 0$ 
measurement. 


3.99 


0.91 


3. 


Applies univariate and bivariate research d^^i&is 
and statistics. 


4.13 


0.80 


4. 


Identifies selected multivariate research &e§i££t\s 
and statistics. 


4.51 . 


0.69 


5. 


Produces or creates a research plan in ccAl^'ho^ 
ation with others. 


4.52 


0.64 


6. 


Implements the research plan collaboratively, 


4.51 


0.73 


7. 


Evaluates participation of self and coll^o** ^-fc^s 
as team members in carrying out the rese^cf 1 
plan. 


4.2P 


0.60 


8. 


Evaluates research studies employing univ^ri-^-fc^ 
and bivariate designs and statistics. 


4.02 


0.84 


9. 


Uses teletype and mini-computer for data '%#-3,y^is 



Section 2. Please assess, on a scale (Likert) of 1 to 5, t^e 



1 = nc 


i; helpfvJ.; 


5 = very helpful. 


3.09 


1.14 


10. 


Text A (required). 


3.85 


1.04 


11. 


Text B (required). 


2.63 


1 .58 


12. 


Supplementary texts. 


4.75 


0.50 


13. 


Handouts on content(i.e. , Research Design* 
Correlation, Inferential Statistics). 


3.89 


0.85 


14. 


Critiques of research studies in class. 


3.98 


0.89 


15. 


Your critique of a research article. 


4.37 


0.80 


16. 


Team research project. 


3.90 


0.97 


17. 


Exams . 


4.72 


0.51 


18. 


Instructor. 
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4.55 0.68 19. Class discussion and lectures 
4.38 0.82 20. Quizzes 



Section 3. Please assess on a scale (Likert) of 1 to 5, the useful- 
ness of each of the following in terms of your learning and your 
being able to fulfill the course requirements"! 1 ~ not at all useful 



5 = very useful. 




J* ( ( 




d 1 • 


Working "by yourself rather than as a team on the 
critinue of thp TPSpareh ai'ticle. 


3.99 


1.21 


22 . 


Working as a research team rather than by yourself 

UX1 i/Ilti X CDCdl Oil JJJ> U J U . 


4.85 


0.43 


23. 


Taking open-book exams rather than closed-book 
exams. 


4.82 


0.51 


24. 


Reviewing descriptive statistics the first day 
of class rather than never reviewing. 


4.21 


0.98 


25. 


Submitting team progress r'eports rather than none. 


4.59 


0.64 


26. 


Getting feedback from your instructor on progress 
reports rather than no feedback. 


4.27 


0.92 


27. 


Critiquing assigned articles in class versus no 
class discussion. 


3.70 


1 .34 


28. 


Written reports rather than class time used for 
reporting of critiques of articles. 



Section 4. Please assess on a scale (Likert) of 1 to 5 $ the following 
aspects regarding your instructor. 1 = poor; 5 = out- 
standing. 



4.92 


0.27 


29. 


Preparation and knowledge of the subject. 


4.80 


0.42 


30. 


Helpfulness to students. 


4.37 


0.70 


31. 


Clarity in explaining concepts and answering 
questions. 


4.55 


0.62 


32c 


Making the course interesting. 


4.31 


0.89 


33. 


Feedback on team progress reports. 


4.62 


0.50 


34. 


Lecture and discussion of course content. 


4.34 


0.76 


35. 


Lecture and class discussion critiquing assigned 
research articles. 


4.56 


0.68 


36. 


Availability for help outside of class. 
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